20. BPTT Quiz 1

_A folded RNN model_

A folded RNN model

Consider the above folded RNN Model. Both states S and Z have multiple neurons in each layer.
The mathematical derivation of state Z at time t is:

SOLUTION: Equation D

Solution

\bar{z} and \bar{s} are vectors, as we indicate that they have multiple neurons in each layer. Using this logic we can understand that equations A and C are incorrect. Since w_2 connects the hidden state \bar{z} to itself, we know that we need to consider the previous timestep here. Therefore only equation D is the correct one.